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Executive Summary 


alfway through the 

2013-14 school year, 
the year No Child Left Be- 
hind (NCLB) envisioned 100 
percent student proficiency, 
school accountability policy 
is more tenuous than ever. 
After years of contesting 
that NCLB was broken and 
that it forced them to iden- 
tify too many schools, or 
the wrong schools, for im- 
provement, over forty states 
have been granted waivers 
from the U.S. Department 
of Education. These waivers 
give them the flexibility to 
design new accountability 
and improvement schemes 
rather than abide by federal 
rules. 
Waivers are not without their rules, con- 
ditions, and guidelines, but they are less 
prescriptive than NCLB. Chiefly, waivers 
encourage school accountability systems 


based on relative, rather than absolute, 
measures of performance. Under NCLB 


accountability, schools identified for im- 
provement fell short of a pre-determined 
performance standard — Adequate Yearly 
Progress (AYP). Under waiver accountability, 
however, the “standard” is often based on 
the number of schools that must be identi- 
fied, rather than their level of performance. 
States must intervene in at least 15 percent 
of their Title | schools (those schools serv- 
ing predominantly low-income students) by 
naming them as priority or focus schools. 
But within this 15 percent framework, states 
can choose how to rank schools, with free- 
dom to design complicated indices and use 
various performance measures, weighting, 
and labels. 


The Department intended for waivers to 
focus school accountability efforts ona 
limited number of schools and improve the 
interventions occurring there. But because 
the interventions under waivers are both 
more focused and more rigorous, the stakes 
for identifying the right schools are higher 
than ever. Waivers can not only alter percep- 
tions of which schools are good or bad, but 
also their chances for getting better. Using 
school improvement data from over 20,000 
schools in 16 states, this report shows how 
the 15 percent framework, and states’ choic- 
es within it, changed the identification of 
schools for improvement during the transi- 
tion from NCLB to waivers. As a result, school 
accountability systems today are dramatically 
different than they were a few years ago. And 
they'll likely stay that way. In the absence of a 
Congressional overhaul of NCLB, waivers are the 
new accountability normal. 
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The 15 Percent 
Framework 


Since states felt that NCLB forced them to name too 
many schools for improvement, many used new, 
relative approaches to accountability to identify fewer 
schools for federal intervention. Across the 16 states, 
the number of schools identified for improvement 
decreased by one-third, on average, from NCLB in 
2011-12 to waivers in 2012-13. But this obscures 
significant variation from state to state: 


e Eleven states classified fewer priority or 
focus schools in 2012-13 than the number of 
schools identified for NCLB improvement in 
2011-12 (AZ, DE, FL, MA, MN, MO, NV, NJ, RI, 
SC, VA). 

e This decrease is much larger in some states 
than in others. Using its waiver, Nevada 
identified over 85 percent fewer schools for 
interventions than it did the previous year 
under NCLB, while Rhode Island identified 12 
percent fewer schools from NCLB to waivers. 

e In five states, however, the number of schools 
identified for improvement reached an all- 
time high under waivers (IN, MS, OK, OR, TN). 
But these states tended to have relatively 
smaller school improvement efforts under 
NCLB. In other words, many of them had to 
identify more schools under waivers in order 
to implement interventions in 15 percent of 
Title | schools in 2012-13. 


The degree of change in the number of schools states 
identify for interventions before and after the tran- 
sition to waivers may show significant variance, but 
states are all moving toward the same target: 15 per- 
cent of Title | schools. Under NCLB, it was much eas- 
ier to make AYP in some states than in others. These 
distinctions matter less with waivers. Take the case of 
Mississippi and Massachusetts: in the 2011-12 school 
year, approximately 12 percent of Mississippi's Title | 
schools were in some stage of improvement, com- 
pared to over 70 percent of Title | schools in Massa- 
chusetts. In the transition to waivers, this variation has 
all but disappeared: every state analyzed has identified 
15 percent of Title | schools as priority or focus, give 
or take a few percentage points. 


States’ Choices 
within the Framework 


States also felt that NCLB’s rigid accountability struc- 
ture prevented them from naming the “right” schools 
for improvement. Accordingly, every state used its 
waiver accountability system to relieve many so-called 


“failing” schools under NCLB from federal interven- 
tions in the 2012-13 school year. 


e Across the 16 states, over 4,400 schools in 
improvement under NCLB were no longer 
identified as priority or focus schools un- 
der waivers. On average, two-thirds of the 
schools identified by NCLB were not among 
the bottom 15 percent of the state’s waiver 
accountability plan. 

e Schools identified for federal interventions by 
NCLB, but not waivers, had been in various 
stages of improvement. In 10 states, the 
majority of schools no longer identified had 
been in the first two years of improvement 
(DE, IN, MN, MS, MO, NJ, OK, OR, TN, VA). 
But in five states, at least half of the schools 
relieved from interventions were previously 
in corrective action or restructuring (AZ, MA, 
NV, RI, SC). 

e Over half of schools previously in restruc- 
turing (those that had missed AYP for six 
consecutive years) were not identified for a 
seventh year of improvement in 2012-13. The 
same is true for over half of schools previous- 
ly in corrective action (those that had missed 
AYP for four or five years). 


Instead of continuing interventions in all previously 
identified schools, waivers allowed states to target 
their efforts on the lowest-performing 15 percent. But 
some states still chose to identify the same schools 
that had been in improvement under NCLB, while oth- 
ers named many priority and focus schools that had 
not been in improvement the year before. 


e _Inseven states, the most common kind of 
priority and focus school is one that had not 
been in interventions under NCLB in 2011-12 
(AZ, IN, MS, OK, OR, SC, TN). And in five, less 
than a quarter of priority and focus schools 
had been in corrective action or restructuring 
during the prior year (AZ, MS, OK, OR, TN). 

e In eight states, the majority of priority and 
focus schools had been in some form of im- 
provement under NCLB in 2011-12 (DE, MA, 
MN, MO, NJ, NV, RI, VA). In five, schools previ- 
ously in restructuring are the most common 
kind of school identified under waivers (MA, 
MO, NV, Ng, RI). 

e States’ accountability design choices may 
affect the kinds of schools they identify in the 
bottom 15 percent. For instance, states using 
super-subgroups based on student achieve- 
ment (instead of demographic characteristics) 
to identify low- 
performing schools tend to name more pri- 
ority and focus schools that had not been in 
improvement in 2011-12. 


The U.S. Department of Education also included two 
safeguards to ensure certain kinds of schools were 
typically priority or focus schools: high schools with 
low graduation rates and schools receiving School Im- 
provement Grant (SIG) awards to implement compre- 
hensive turnaround efforts. But these safeguards did 
not necessarily result in a greater emphasis on high 
school accountability or the consistent identification 
of SIG schools. 


e Eight states increased the proportion of 
their schools in improvement that were high 
schools, but many of these changes were rel- 
atively small (IN, MA, MN, MS, NV, NJ, OR, TN). 

e The increase in high school accountability 
in these states typically came at the expense 
of middle schools, rather than elementary 
schools. 

e In twelve states, nearly every SIG school is 
identified as a priority or 
focus school under waiver accountability sys- 
tems, and in three, 100 percent of SIG schools 
are priority schools (MN, RI, VA). But in three 
others, 50 percent or more of SIG schools are 
not identified at all (IN, NV, SC). 


Above all, waivers are best understood in context 

of the shift toward relative rankings for school ac- 
countability. Because of the switch to a 15 percent 
framework, thousands of schools were eased from 
federal interventions in a single school year. And since 
states could determine how to measure their relative 
performance, waivers also tend to increase the vari- 
ation between states. Some states used their waivers 
to redouble efforts to improve previously identified 
schools, while others took the opportunity to change 
which schools receive scarce school improvement 
resources and supports. 


Moving Forward 


There may be agreement that NCLB is broken, but 
there is not yet agreement that waivers are working. 
This is in large part because so little is known about 
waiver implementation and its effect on schools and 
student learning. The data presented in this paper are 
some of the first to show the differences in school 
accountability between NCLB and waivers. But many 
questions remain. Further research is sorely need- 
ed, especially as the U.S. Department of Education 
begins to review and renew states’ waivers during the 
2013-14 school year. The renewal process would be 
a natural opportunity to correct course and improve 
waiver policy. But not enough data have been gath- 
ered to determine what these changes should be. 
While it is understandable that the Department may 
need additional time and manpower to gather and 
analyze waiver data, it must continue these efforts and 
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make the information available to researchers, ana- 
lysts, journalists, policymakers, and practitioners. Giv- 
en the critical need for better data on NCLB waiver 
implementation, the Department should reconsider 
and strengthen, where necessary, how it plans to 
provide ongoing oversight of waivers, including 
the renewal process, to ensure it is comprehensive, 
transparent, and responsive. 


There are three other considerations for the Depart- 
ment and states as they collect data on waiver imple- 
mentation, conduct research on the effectiveness of 
states’ plans, and revise or renew the waiver requests. 
These efforts should consider: 


Improving the alignment between states’ 

school goals, school quality measures, 
and the process for identifying priority and 
focus schools. 


Increasing states’ school improvement 

capacities and altering the methodolo- 
gies for naming priority and focus schools 
to ensure certain kinds of schools are always 
identified, especially if the schools that go 
unidentified show less improvement over 
time. 


Changing states’ school performance 

measures or accountability design choic- 
es if there is a link between specific choices 
and better outcomes for schools or students. 


With waivers, states have the opportunity to rethink 
how they identify and improve their lowest-perform- 
ing schools. And while waivers are clearly compli- 
cated, so was NCLB. The problem with No Child Left 
Behind wasn't that it was complicated, however. The 
problem was that the measures and methods it used 
to identify low-performing schools weren't very effec- 
tive. Unfortunately, it isn't clear if states and the U.S. 
Department of Education have learned these lessons 
and applied them to waiver accountability systems. 
Identifying low-performing schools is the easy part, 
compared to actually improving them. As states begin 
to renew their waivers and make their case anew to 
the Department, it's not just time for more plans and 
promises — it’s time to finally get school accountabili- 
ty systems right. 
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Introduction 


or as long as there have 

been public schools in 
the United States, there have 
been efforts to make them 
better. American school re- 
formers have taken on tru- 
ancy, overcrowding, segre- 
gation, and more recently, 
low academic standards. 
But through each wave of 
reform, one thing has re- 
mained constant: there have 
always been bad schools — 
and minorities, immigrants, 
and children in poverty or 
with special needs have 
been much more likely to 
attend them. 


These schools are no secret. They have 
gone by many names: slum schools, 
failing schools, and dropout factories, 

to name a few. But they became much 
more conspicuous thanks to the No Child 
Left Behind Act of 2001 (NCLB). NCLB, 
the latest iteration of the Elementary and 
Secondary Education Act (ESEA), sought 
to close the achievement gap so that all 
students would be proficient in reading 
and math by 2014, the current school 
year. To that end, the law required states 
to adopt not only academic standards 
and annual tests for all students, but 
also performance goals and mandated 
consequences for schools that failed to 
make AYP, or Adequate Yearly Progress 


(see ‘AYP and School Improvement Un- 

der NCLB’). More important, test results 
were broken down by key subgroups of 
students, and performance targets were 
applied to these sub-populations within 
schools. 


Thus, NCLB systematically identified 
low-performing schools and forced 
states and school districts to do some- 
thing about them. And for the first time, 
states began to collect and publish large 
amounts of student achievement data to 
track public school performance. Poli- 
cymakers, school reformers, and parents 
always knew bad schools existed. Now 
they had hard evidence to back it up. 


Flash forward a decade, however, and 
this system of school accountability has 
been quietly dismantled and rebuilt. 
States, armed with waivers that give 
them newfound flexibility from the 
federal government, have designed 
their own approaches to identify and 
improve low-performing schools. With 
NCLB six years overdue for a rewrite and 
no legislative solution in sight, waivers 
have become the new normal. While the 
resurgence of state-driven accountabil- 
ity systems has allowed for policy inno- 
vation in the states, it has also promoted 
policy incoherence nationally, adding 
complexity and making information 

on states’ choices difficult to come by. 
Over time, state-driven accountability 
systems may prove to be more effective 
than NCLB, but little research has been 
conducted on their implementation. 
This paper provides some of the first 
evidencedence of how schools facing 
improvement changed between NCLB 
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and waiver accountability systems, using data from 
over 20,000 schools in 16 states transitioning from the 
last year of NCLB (2011-12) to the first year of waivers 
(2012-13). 


One of these schools is in Michigan City, Indiana — a 
city of 31,000 wedged between Gary and South Bend 
on the shore of Lake Michigan. Knapp Elementary 
School was perennially on the list of schools failing 
to make AYP. Its student body reflects the struggles 
of the post-industrial city as a whole: it is located just 
over a mile from the maximum-security Indiana State 
Prison, and over 85 percent of its students are low-in- 
come. On the state’s school report card, Knapp was 
almost always a “C" pupil, although the school had 
shown signs of improvement. Still, after the 2010-11 
school year, Knapp failed to make AYP yet again, and 
the school began its sixth year of improvement in the 
fall of 2011. This meant Knapp was placed in some of 
the most rigorous interventions offered under NCLB. 


But just over a year later, Knapp Elementary was no 
longer “failing.” While the school hadn't improved 
from its “C" grade, it was no longer subject to federal 
school interventions. Indiana had been given the go- 
ahead from the U.S. Department of Education to use 
a new accountability system, based on Indiana’s state 
report cards rather than the stamdards set by NCLB. 
The new accountability system aimed to concentrate 
interventions on the schools that were struggling the 
most, and Knapp didn’t make the cut. 


There are likely thousands of schools like Knapp 
across the country. Over forty other states also have 
waivers from NCLB, and with them, the flexibility to 
design new school accountability and improvement 
schemes rather than abide by federal rules. NCLB’s 
rules, states argued, forced them to identify too many 
or the wrong schools for improvement. 


Waivers laid out a few new rules for states, but as 

a whole, they were far less prescriptive than NCLB. 
States wanted systems that were more responsive to 
their local needs and that better targeted resources to 
the most critical areas. Importantly, waivers opened 
the door for school accountabilty based on relative, 
rather than absolute, school performance. Under 
NCLB accountability, schools identified for improve- 
ment fell short of pre-determined performance 
standards. Under state-driven waiver accountability, 
however, the “standard” is often based on the number 
of schools that must be identified, rather than their 
performance. States must intervene in at least 15 
percent of Title | schools (those serving predominant- 
ly low-income students): the bottom 5 percent are 
named “priority” schools and another 10 percent are 
identified as “focus” schools, typically because of low 
subgroup achievement or large achievement gaps. 
States filled in the remaining details on their own, with 
latitude to choose the goals, measures, weighting, and 
school labels they prefer. Indiana used this flexibility 


to incorporate bonuses for student growth and give all 
schools an A-F grade, but other states made different 
choices. As a result, the existence of priority and focus 
schools is often the only common element across state 
accountability systems today — and it’s the only one that 
always carries meaningful stakes for schools. 


But the stakes for states to identify the right schools are 
also higher, since the interventions under waivers are 
more rigorous and more focused on a limited number 
of schools. Waivers can not only alter perceptions of 
which schools are good or bad, but also their chanc- 

es for getting better. With waivers, schools that ap- 
peared to be just fine under NCLB may become priority 
or focus schools, while some of NCLB's chronically 
low-performing schools go unidentified. Newly iden- 
tified schools will face additional public scrutiny and 
requirements, altering the responsibilities and expecta- 
tions placed on educators there. This is especially true 
for priority schools, which must enter a comprehensive 
three-year turnaround program with stricter require- 
ments than NCLB (see “Turnaround Principles for Priori- 
ty Schools”). Meanwhile, schools identified under NCLB, 
but not waivers, may discontinue some interventions 
after losing the supplemental resources they received 
based on NCLB identification. But there is little evidence 
of whether this occurred, because little is known about 
whether the schools identified for interventions have 
changed and how. 


This analysis demonstrates that school accountabili- 

ty systems are changing under NCLB waivers, but the 
effect of the new federal rules and states’ choices within 
them varies tremendously from one state to another. 
Most states identify fewer schools for improvement un- 
der waivers, but a few do not. Some continue to focus 
their school improvement efforts on the same schools 
that were considered low-performing under NCLB, 
while other states have intervened in schools that were 
not identified the year before. And it’s not always clear 
how the various choices states made work together to 
determine which schools are identified and which are 
not. 


But one thing is clear: Across the board, without ex- 
ception, some of the worst schools according to NCLB 
are no longer identified for improvement under NCLB 
waivers. The untold story of waivers is that with little 
warning, hundreds of so-called “failing” schools sud- 
denly weren't “failing” anymore. Why? These schools 
may have been misidentified by NCLB, but the more 
likely reason is the new federal approach for identifying 
the worst schools: the 15 percent framework. A relative 
accountability strategy creates a finite number of school 
improvement slots within states. At the same time, 
states were given flexibility to measure school perfor- 
mance differently, changing perceptions of their relative 
success. While it may be too soon to tell if 15 percent is 
the “right” number or if states have chosen the “right” 
performance measures, landing in the bottom 15 per- 
cent of schools in a state matters for teachers, students, 
and families, and it matters now. 


AYP & School Improvement 
Under NCLB 
Years NCLB-Prescribed Intervention for 


Missing Schools In Need of Improvement 
AYP 


1 None 


2 School Improvement Year 1: District 
must offer students option to transfer to 

3 School Improvement Year 2: District 
must also offer supplemental education 
services (SES) to students. 


4 Corrective Action Year 1: In addition to 
the transfer option and SES, schools 
must take larger steps to improve the 
school, such as changes in staff or lead- 
ership, curriculum reforms, increasing 
or shifting instructional time, or working 
with external providers to improve. 


5 Corrective Action Year 2: School contin- 
ues transfer option, SES, and corrective 
actions and creates a restructuring plan. 


6 Restructuring: School implements 
restructuring plan, which could involve: 
replacing principal and some staff, 
state takeover, conversion to a charter 
school, contracting with another orga- 
nization to manage the school, or other 
changes in leadership and governance. 
Most schools chose the ‘other’ model, 
the most flexible and permissive option.’ 


‘See, for example, Sara Mead, “Easy Way Out: “Restructured” 
usually means little has changed,” Education Next, Winter 
2007, http://educationnext.org/files/ednext_20071_52.pdf, 
and Robert Manwaring, “Restructuring ‘Restructuring, Edu- 
cation Sector, 2010, http://www.educationsector.org/sites/ 
default/files/publications/Restructuring. pdf. 


Turnaround Principles for 
Priority Schools 


All priority schools — the bottom 5 percent of Title | 
schools in the state — must implement a three-year 
turnaround strategy to improve academic achieve- 
ment. 


This strategy must align with key turnaround princi- 
ples identified by the U.S. Department of Education. 
The four turnaround models adopted by schools in 
the School Improvement Grants program meet all of 
these principles. 


States may also use a statewide approach that meets 
the principles, including transferring control of 
school operations to the state, to a recovery school 
district, or to another management organization. 
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All priority schools must adopt interventions 
that meet key principles. These are: 


@ Providing strong leadership by: (i) re- 

mis Viewing the performance of the current 
principal; (ii) either replacing the principal if 
such a change is necessary to ensure strong 
and effective leadership, or demonstrating 
to the SEA that the current principal has a 
track record in improving achievement and 
has the ability to lead the turnaround effort; 
and (iii) providing the principal with opera- 
tional flexibility in the areas of scheduling, 
staff, curriculum, and budget. 


# Ensuring that teachers are effective and 
m able to improve instruction by: (i) re- 
viewing the quality of all staff and retaining 
only those who are determined to be effec- 
tive and have the ability to be successful in 
the turnaround effort; (ii) preventing inef- 
fective teachers from transferring to these 
schools; and (iii) providing job-embedded, 
ongoing professional development informed 
by the teacher evaluation and support sys- 
tems and tied to teacher and student needs. 


=¥ Redesigning the school day, week, or 
year to include additional time for stu- 
dent learning and teacher collaboration. 


| Strengthening the school’s instructional 
@ program based on student needs and 
ensuring that the instructional program is 
research-based, rigorous, and aligned with 
State academic content standards. 


Using data to inform instruction and for 
wW continuous improvement, including by 
providing time for collaboration on the use 
of data. 


™ Establishing a school environment that 

w@ improves school safety and discipline 
and addressing other non-academic fac- 
tors that impact student achievement, such 
as students’ social, emotional, and health 
needs. 


"#7 Providing ongoing mechanisms for fami- 
# lyand community engagement. 
of Education, June 7, 2012 (http://www.ed.gov/esea/flex- 
ibility/documents/esea-flexibility-acc.doc accessed June 
22, 2013). 
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The Case for Flexibility 


or schools like Knapp, 

there is a huge difference 
between being labeled a “C” 
school on the state's report 
card and being identified as 
a priority or focus school. 
Public reporting of school 
performance data is well 
and good for transparen- 
cy’s sake, but the purpose 
of school report cards is to 
detect, rather than fix, the 
problems in low-perform- 
ing schools. Unlike “report 
card’ forms of accountabil- 
ity, however, identification 
as a priority or focus school 
goes further. The stakes are 
high, particularly for prior- 
ity schools, because of the 
sanctions, interventions, or 
significant changes in gov- 
ernance that are supposed 
to accompany the label. In 
priority schools, families 
may be able to send their 
children to other schools, 
educators could lose their 
jobs, and schools could gain 
supports and resources, be 


placed under new leader- 
ship, reopen as a charter, or 
even close. Without signif- 
icant changes in leadership 
and governance, low-per- 
forming schools will likely 
remain that way. 


NCLB is no longer considered the best 
vehicle to produce those changes (see 
“NCLB: Failing to Identify and Improve 
‘Failing’ Schools”). Over a decade into 

its implementation, achievement gaps 
persist, dropout factories continue to 
operate, and students routinely graduate 
high school unprepared for postsec- 
ondary training and work. Research has 
found that NCLB’s accountability system 
has led to some gains in school math 
performance, particularly in elementary 
grades, but found no effect on reading 
performance.‘ This mirrors trends in 
student performance on the Nation- 

al Assessment of Educational Progress 
(NAEP). In the last decade, math scores 
have improved somewhat, while 

reading scores have mostly remained 
steady.* These lackluster results, far from 
NCLB’‘s universal proficiency goal, and 
the problems with NCLB’‘s accountability 
and improvement strategies have fueled 
discontent with the law. 


No Child Left Behind is now six years 
behind schedule for reauthorization. And 
as state reforms butt against the outdat- 
ed federal system, many feel NCLB con- 
strains states from fully embracing new 
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NCLB: Failing to Identify and Improve “Failing” Schools 


No Child Left Behind identifies the worst schools based 
on simple and static measures, like proficiency rates 
on standardized reading and math tests, which can be 
easily manipulated by states. States determine what 
cut scores are necessary for students to be deemed 
proficient, the precision with which proficiency rates 
are measured, and what kinds of credentials count as 
completing high school.' 


These arbitrary decisions within the accountability 
criteria create perverse incentives. Seemingly minor 
tweaks, like reducing the number of students required 
in a subgroup for accountability purposes from 20 

to 15, could make the difference between a school 
making AYP or not. Further, NCLB’'s end-of-the-year 
snapshot does not recognize schools whose students 
demonstrate impressive growth from year to year, 
regardless of their proficiency." It also fails to acknowl- 
edge other measures of school success, including 
highly effective teaching and whether students are 
adequately prepared for college and careers. 


NCLB’s problems extend beyond how it identifies 
low-performing schools. The law's methods for im- 
proving them are also flawed. It requires a standardized 
series of school interventions, based on the number of 
years a school failed to meet its performance targets 
rather than why it failed to meet them. A school that 

is identified because its special education students are 
overwhelmingly below grade level in reading is treated 
exactly the same as a school where almost all of its 
students are low-performing readers. And as NCLB’s 
performance targets inched toward universal profi- 
ciency, more and more schools fell into improvement 
each year, stretching thin state and local capacity to 
improve them. Kentucky Commissioner of Education 
Terry Holliday echoes these criticisms: “Our trouble 

is we identify too many schools for intervention. Our 
most difficult challenge is to differentiate the level of 
support the state provides to districts and schools.” 


In the worst cases, states eased the targets or stan- 
dards rather than engage in NCLB’s regimen of im- 
provement activities for more and more schools.” 


The standardized interventions also didn’t produce the 
dramatic kind of improvement NCLB proponents en- 
visioned. One evaluation found that only one percent 
of eligible students took advantage of NCLB's choice 
option.” A greater share of students participated in tu- 
toring programs, but their effects on student achieve- 
ment were limited and inconsistent, and the quality 
and business practices of many tutoring companies 
were dubious.” 


The U.S. Department of Education's Inspector General 
has had to respond to an increase in fraud and corrup- 
tion cases against tutoring providers, and even former 
Department of Education officials, like Michael Petrilli 
of the Thomas B. Fordham Institute, admit the program 
was “an unmitigated disaster."“’ And the worst schools 
— those in corrective action or restructuring — rarely 
made dramatic changes. In 2007-08, over three-quar- 
ters of schools in restructuring took advantage of wide 
latitude to design “other,” and often less intensive, 
strategies.“ 


‘These criticisms underpin many states’ decisions to 
apply for waivers. The opening lines of Massachusetts’ 
approved waiver accountability system are typical: 


The Commonwealth's schools and districts are cur- 
rently assessed based on both the state's five-level 
Framework for District and School Accountability 

and the requirements of the No Child Left Behind Act 
(NCLB). Operating these dual systems at one time pro- 
vided a wealth of valuable feedback, but the require- 
ments under NCLB have declined into an adminis- 
trative and fiscal burden that is no longer useful. The 
rising targets have resulted in far too many schools 
and districts being identified as in need of improve- 
ment to allow the state to best identify those most 
needing assistance or intervention. 


i See, for example, Kevin Carey, “The Pangloss Index” Education Sector, 
2006. http: Jw. educationsector.org/sites/default/files/publications/ 


eine aren EAhisAtluploate WhesColnd Bey 
ii NCLB includes a “safe harbor” provision which allows a school to 
make AYP via improvement ial one, rates without meeti ce state 


percent from one year to next, even if that saa ip missed its AMO. 
For more, see Erin Dillon and Andrew J. Rotherham, “States’ Evidence: 
What It Means to Make ‘Adequate Yearly Progress’ Under NC 
Education Sector, 2007 http://www.educationsector. org/sites/default/ 
files/publications/EXPAYP. pdf. 

iii Michele McNeil, “NCLB Waiver-Renewal Process Turns Up Heat on 
States,” Education Week, September, 10, 2013 http://www.edweek.org/ 
ew/articles/2013/09/11/03waiver_ep.h33.html (accessed October 24, 
2013). 

iv Bandeira de Mello, V. (2011), Mapping State Proficiency Standards 
Onto the NAEP Scales: Variation and Change in State Standards for 
Reading and Mathematics, 2005-2009 (NCES 2011-458). National 
Center for Education Statistics, Institute of Education Sciences, U.S. 
Department of Education, Washington, DC: Government Printing 
Office. 


v Gill, Brian, Jennifer Sloan McCombs, Scott Naftel, Karen E. Ross, 
Mengli Song, Jennifer Harmon, Georges Vernez, Beatrice Birman, 
Michael Garet and Jennifer O' Poe State and Local Implementation of 
the No Child Left Behi ct | Choi e and 
Supplemental Educa a 
RAND Corporation, 2008. ee. pies rand: seneoty ena ints/RP1332. 
vi Carolyn J. Heinrich and Patricia Burch, “The Implement n and 
Effectiveness of Supplemental Educational Services,” American 

tu Center for American Progress, March 2012 
http: ‘/www.aei. org/files/2012/03/05/- \e-implementation-and-ef- 
fectiveness-of-supplemental-educational-services_17150915643.pdf 
(accessed November 29, 2013). 

vii See, William D. Hamel, “Final Management Information Report Fraud 
in Title |-Funded Tutoring Programs,” Office of Inspector General, U.S. 
Department of Education, October 31, 2013 http://www2.ed.gov/ 
about/offices/list/oig/auditreports/fy2013/x42n0001.pdf (ac: 
November 19, 2013), and Morgan Smith, “After Misuse, a Push tc 
tinue Tutoring Mandate,” The Texas Tribune, October 20, 2013 ht 
www.texastribune.org/2013/10/20/after-misuse-push-continue-tutor- 
ing-mandate/ (access October 24, 2013). 

viii Robert Manwaring, “Restru ing ‘Restructuring, Education 
Sector, 2010, http://www.educationsector.org/si i 
cations/Restructuring.pdf. 


approaches to better identify, support, and improve 
low-performing schools. In testimony before Con- 
gress, New York Commissioner of Education John 
King explained, “New York was finding it increasingly 
difficult to keep its ESEA accountability system well 
aligned with the bold agenda for educational reform 
that the Board of Regents established.” And it is telling 
that even in today’s hyper-partisan political environ- 
ment, nearly everyone agrees NCLB no longer works. 
As Denver schools’ superintendent-turned U.S. Sena- 
tor Michael Bennet put it, “if you had a rally 

tomorrow on the Capitol to keep No Child Left Behind 
the same, there's not a single person that would show 
up for that rally.”4 


The U.S. Department of Education has tried to re- 
spond to states’ needs. During the second half of the 
Bush administration, the Department offered two 
pilot programs, one to experiment with measures of 
student growth and another to develop “differentiated 
accountability” systems.® Bipartisan state advocacy 
groups, including the Council of Chief State School 
Officers, National Governor's Association, and Nation- 
al Association of State Boards of Education, embraced 
both strategies as part of NCLB reauthorization.® So 
did the Obama administration in its reauthorization 
proposal, A Blueprint for Reform.’ But NCLB waivers 
represent the strongest, and largest, endorsement for 
new approaches to identify and improve America’s 
worst schools to date. 


NCLB Waivers: 
New Game, New Rules 


By using frustration with NCLB and building on pre- 
existing state efforts, the U.S. Department of Educa- 
tion was able to entice all but four states — Montana, 
Nebraska, North Dakota, and Vermont — to apply for 
flexibility from NCLB on the condition that they would 
implement reform in four key areas (see “Waiver Nuts 
and Bolts”).2 Within a year of the first application due 
date, 34 states and Washington, D.C. had won flexibil- 
ity from the Department of Education. Now, 42 states, 
Washington D.C., Puerto Rico, and a consortium of 
eight California school districts have been awarded 
waivers. Most of these waivers expire at the end of the 
2013-14 school year, but states will be able to renew 
them for an additional year. And if NCLB reautho- 
rization continues to languish in Congress, further 
renewals are likely. 


States frequently criticized NCLB’s rigid accountability 
structure for conflicting with state-level reforms and 
for forcing them to identify too many or the “wrong” 
schools for improvement. In response, the Depart- 
ment’s waiver guidelines only require federal interven- 
tions in 15 percent of Title | schools, allowing states 
to more narrowly focus and strengthen improvement 
efforts. The Department had already bet on this tactic 
when it revamped the School Improvement Grant 
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Program in 2009, infusing unprecedented levels of 
funding ($3 billion) into the program and upping the 
intensity of interventions in hundreds of the nation’s 
chronically low-performing schools. The Department 
defined these schools similarly to priority schools: the 
bottom 5 percent of schools based on student profi- 
ciency and progress. SIG schools choose from four 
improvement strategies, each of which 

requires more specific reforms than NCLB. 


No Child Left Behind may not have been rewritten, 

but thanks to waivers, most of the law's accountability 
provisions are obsolete. As Oklahoma Superintendent 
of Education Janet Barresi put it, flexibility “is a game 
changer. ... We now have added urgency to press ahead 
with the implementation of reforms and a chance to 
help schools in our state improve.”® For the foreseeable 
future, the majority of states will be using their own 
accountability and school improvement mechanisms 
rather than those prescribed in federal law. 


For educators, local administrators, and parents, waiv- 
ers could mean that their school will no longer be in 
improvement: it will be a “C” school, or a focus school, 
or not labeled at all. Families will receive new school 
report cards too, with different information about the 
performance of their local school. Given these chang- 
es, teachers, families, policymakers, and other stake- 
holders desperately need information to understand 
what waivers mean. But information is hard to come 
by. After ten years, NCLB, AYP, and its school improve- 
ment steps are familiar. Waivers introduce a whole new 
accountability language (see “Comparing NCLB and 
Waiver Accountability Systems”). And it does not help 
that states’ waivers tend to be hundreds of pages, full 
of jargon and technical details, and based on inter- 
locking, complicated reforms. Further, when parents, 
educators, and advocates do successfully navigate their 
state's waiver request, they often become upset at the 
changes envisioned for school accountability systems, 
arguing that they set different expectations for minori- 
ty students, or unfairly treat certain kinds of schools.’ 
These concerns are shared by many national education 
and civil rights advocacy organizations, including the 
Education Trust, Center for American American Prog- 
ress, Alliance for Excellent Education, and Campaign 
for High School Equity." 


Waivers have changed the school accountability game, 
but does anyone know the rules? Without clear and 
accurate information about the impact of states’ new 
waiver accountability systems, it is practically impossi- 
ble for the public to judge the merits of states’ efforts. 
How have the components of states’ accountability 
systems changed, including the measures and labels 
used to describe school performance? And more 
important, how do these changes affect the group of 
schools feeling the consequences of the new account- 
ability systems — those placed in improvement as 
priority and focus schools? 


13 


14 It’s All Relative 


Waiver Nuts 
and Bolts 


In September 2011, President Obama and 
Secretary Duncan unveiled a plan to offer 
all states flexibility from No Child Left 
Behind, if they met certain conditions. With 
Department approval, states could receive 
two-year waivers to change their perfor- 
mance targets and school accountability 
systems in exchange for reforms in four 
areas: college- and career-ready expecta- 
tions, differentiated accountability sys- 
tems, effective instruction and leadership, 
and reduction of unnecessary reporting 
and administrative requirements.' The first 
states were awarded waivers in February 
2012, followed by two additional rounds of 
requests. States from the first two applica- 
tion rounds that received waivers for the 
2012-13 and 2013-14 school years are now 
in the process of renewing these requests 
for an additional year. 


NCLB Waivers in the States 


HEME No Waiver (CA state, MT, NE, ND, VT) 


Round 1 (CO, FL, GA, IN, KY, MA, MN, 
NJ, NM, OK, TN) 


Waiting for Approval for Round 2 
(IA, IL) 

Round 2 (AZ, AR, CT, DC, DE, ID, KS, LA, MD, MI, MS, 
MO, NV, NY, NC, OH, OR, RI, SC, SD, UT, VA, WA, WI) 
Waiting for Approval for Round 3 

(BIE, WY) 


Round 3 (AL, AK, HI, ME, NH, PA, 
PR, TX, WV, 8 CA Districts - see insert) 


High Risk States (AZ, OR, KS, WA) 


N it 


California District Waivers: 
Eight unified school districts 
in California (Los Angeles, 
Long Beach, San Francisco, 
Oakland, Santa Ana, Sanger, 
Sacramento City and Fresno) 
received district waivers 
under the No Child Left Be- 
hind Act in August 2013. 


Principles of ESEA Flexibility 


College- and Career-Ready Expectations for All 
Students. States must: 


e adopt college- and career-ready standards in 
English language arts and mathematics; and 
administer high-quality, aligned assessments that 
measure student growth in grades 3-8 and at 
least once in high school. States must also adopt 
English Language proficiency standards and 
related assessments, as well as assessments for 
students with disabilities. 


Magnified: 
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State-Developed Differentiated Recogni- 
tion, Accountability, and Support. States 
must: 


e develop differentiated accountability sys- 
tems that use both achievement and prog- 
ress data to measure school performance, 
and establish annual goals for student 
performance; 

e identify Title | schools with high progress 
and/or achievement as “reward schools” 
and offer them public recognition or other 
incentives; 

e identify at least 5% of Title | schools 
with the lowest performance as “priority 
schools” to undergo comprehensive three- 
year turnaround efforts; and 

e identify at least 10% of Title | schools with 
large achievement gaps and/or low sub- 
group performance as “focus schools” and 
support them through a targeted school 
improvement process. 


Supporting Effective Instruction and Lead- 
ership. States must: 


e develop, pilot, and implement new teacher 
and principal evaluation systems that 
include a student growth component, 
differentiate teachers into multiple cate- 
gories of effectiveness, provide feedback 
for professional development, and inform 
personnel decisions. 


Reducing Duplication and Unnecessary 

Burden. States must:evaluate their admin- 
istrative requirements and reduce duplicative 
reporting and unnecessary burden on school 
districts and schools, eliminating regulations 
that have little impact on student perfor- 
mance. 


' For further information and related documentation, see U.S. Department of Education's ESEA Flexibility page: http://www2. 
ed.gov/policy/elsec/guid/esea-flexibility/index.html. 
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Comparing NCLB and 
Waiver Accountability Systems 


Accountability 
Component 


Student 
Performance 
Goals 


Relationship 
between Goals 
and School 
dentification 


School 
Identification 


School 
Improvement 


NCLB 
Accountability 


State sets annual measurable 
objectives (AMOs) that en- 
sure all students and student 
subgroups achieve 100% 
proficiency by 2014. 


AMOs are used to determine 
if schools make Adequate 
Yearly Progress (AYP) for all 
students and student sub- 
groups. 


Title | schools failing to make 
AYP for any subgroup for two 
consecutive years are iden- 
tified as schools in need of 
improvement. 


Title | schools in need of im- 
provement implement inter- 
ventions based on the num- 
ber of years they have missed 
AYP. To exit improvement, 
states must make AYP for two 
consecutive years. 
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Waiver 
Accountability 


State sets annual measur- 
able objectives (AMOs) for 
all students and student 
subgroups, but choose the 
end goal: 100% proficiency, 
decreasing achievement 
gaps by 50%, or another 
outcome. 


AMOs for all students and 
student subgroups are used 
in states’ differentiated 
accountability systems and 
may be used to identify 
priority and focus schools. 


At least 5% of the low- 
est-performing Title | 
schools are identified as 
priority schools; at least 
10% of Title | schools with 
large achievement gaps 
and/or low subgroup per- 
formance are identified as 
focus schools. States choose 
how to use AMOs, student 
subgroups, and other per- 
formance measures to make 
these determinations. 


Priority schools implement 
three-year interventions 
that reflect key turnaround 
principles; focus schools 
implement sanctions target- 
ed to their focus area. States 
determine criteria for exiting 
improvement. 
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School Accountability 2.0 


T o examine how school 
accountability systems 
shifted from NCLB to waiv- 
ers, this report analyzes 
2011-12 and 2012-13 
school improvement data 
and waiver requests from 16 
states: Arizona, Delaware, 
Florida, Indiana, Massachu- 
setts, Minnesota, Mississip- 
pi, Missouri, Nevada, New 
Jersey, Oklahoma, Oregon, 
Rhode Island, South Caroli- 
na, Tennessee, and Virginia. 


We selected these states to be as repre- 
sentative as possible of the 35 implement- 
ing flexibility during the 2012-13 school 
year.” In each state, data were collected 
on the schools identified for improvement 
under No Child Left Behind in 2011-12 

as well as schools identified as priority or 
focus schools in 2012-13. The data in- 
clude whether the school was identified 
for improvement, the label assigned (e.g., 
corrective action, priority), and other char- 
acteristics including the school’s grade 
level, whether it was a charter or tradi- 
tional public school, and whether it was 
eligible for or received School Improve- 
ment Grant funding. Data were collected 
from state education agency websites, and 
when possible, include not only schools 
identified for improvement in a given year, 
but also the entire population of schools 
in the state. In sum, over 20,300 schools 
are represented in the dataset across the 


16 states. 


Some states chose to create additional 
categories of schools to “watch” beyond 
priority and focus schools, although it 
was not required by the U.S. Department 
of Education. States are also monitoring 
the performance of all Title | schools 
against performance goals, regardless of 
the school’s priority or focus status. But 
the interventions in these schools are 
not nearly as meaningful or ambitious 
as those in priority and focus schools. 
They may be placed under additional 
district or state oversight, but it is not 
equivalent to the resources, support, 
and attention priority and focus schools 
receive from their districts, states, and 
the federal government. Thus, only pri- 
ority and focus schools are considered 
to be facing federal improvement inter- 
ventions under waiver accountability in 
this analysis. 


Just as it is difficult to compare the 
effects of NCLB across states because 
each used different academic standards, 
assessments, proficiency cut scores, and 
annual performance targets, it is also 
difficult to compare the effect of waiv- 
ers across states. Within the guidance 
provided by the Department, states had 
autonomy to choose how to measure 
school performance; how to weight 
each measure within an overall index, 
grade, or school label; and how to 
identify schools to receive interventions. 
Because of these additional choices, 
waivers have tended to increase the 
variation between the design of states’ 
accountability systems, and often, the 
results they produce (see Appendix A: 
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Summary of Waiver Accountability Design Choices). 
This analysis focuses on changes within states be- 
tween NCLB and waiver accountability. While some 
overarching lessons can be learned from the data, the 
amount of flexibility waivers offered to states limits 
the number of general conclusions that can be drawn 
about these changes nationally. 


NCLB Schools in Improvement: 
In or Out? 


Waivers gave states significant latitude in how they 
identify schools for improvement, and states took 
advantage of it. In the 16 states here, the group of 
schools targeted for interventions under waivers is 
often radically different from NCLB. As Table 1 demon- 
strates, in nine states, over two-thirds of all Title | 
schools identified for improvement under NCLB were 
no longer identified in waiver accountability systems 
— the Knapp Elementary Schools of the world. And in 
all but two states, Indiana and Rhode Island, over half 
of schools receiving interventions under NCLB were 
not labeled as priority or focus schools. 


That is not to imply that NCLB’s method for identify- 
ing “failing” schools was perfect. In fact, many states 
added new metrics to their waiver accountability plans 
to improve school identification, including student 
growth and college- and career-ready indicators. But 
despite the ability to try new accountability approach- 
es with waivers, proficiency rates continue to play a 
significant role, even a starring one, in states’ identi- 
fication of priority and focus schools."* Thus, certain 
schools would be expected to be removed from fed- 
erally required interventions more often than others 
under waivers: schools that had barely missed AYP and 
schools in the least severe stages of improvement. A 
school that was implementing restructuring (NCLB’s 
intervention for chronically underperforming schools) 
should have a higher likelihood of also being identified 
with waiver accountability systems in force. These 
schools had failed to meet their performance targets 
not once, not twice, but six consecutive times. 


As Figure 1a shows, these expectations are confirmed 
in most states: when schools in NCLB improvement 
are not named as priority and focus schools, they tend 
to come from the least serious NCLB intervention 
phases.” In 10 states, at least six of every 10 schools 
removed from federal interventions through waiv- 

er accountability had been in the first two years of 
school improvement under NCLB. And in all but two, 
fewer than one of every 10 schools removed from 
interventions had been in improvement for over six 
years. 


But there are also states where a larger proportion of 
schools eased from federal interventions in the tran- 


sition to waivers come from the later, more serious 
phases of NCLB improvement (Figure 1b). In Arizona, 
Massachusetts, Nevada, Rhode Island, and South 
Carolina at least half of all schools removed from 
federal improvement had been in corrective action or 
restructuring under NCLB. And in South Carolina, half 
of schools let off the hook had been in restructuring. 


There is an important caveat to these data. The dis- 
tribution of schools removed from federal account- 
ability under waivers must be considered in light of 
how interventions were distributed in these states 
under NCLB. As Table 2 shows, the three states where 
schools in restructuring represent only a fraction of 
the schools eased from improvement (Delaware, Mis- 
sissippi, and Oklahoma) are also the three states with 
the smallest share of schools in restructuring to begin 
with. And each of the five states where schools in 
restructuring make up a large share of those removed 
from federal interventions under waivers (Arizona, 
Massachusetts, Nevada, Rhode Island, and South Car- 
olina) had a higher-than-average share of schools in 
restructuring than other states sampled. 


It is also clear, however, that some states used their 
waiver accountability systems to refocus efforts to 
improve schools in restructuring, while others were 
more likely to exclude them from their priority and fo- 
cus schools. Massachusetts, Missouri, and New Jersey 
took the former approach. 


While many of their schools in restructuring are no 
longer identified, the proportion of schools in im- 
provement that are or were in the restructuring 
phase increases significantly from NCLB to waiver 
accountability. Nearly half of Massachusetts’ schools 
in improvement in 2011-12 were in restructuring, 

but in 2012-13 over 80 percent of schools in im- 
provement were in restructuring the previous year. 

In other words, few of their priority and focus school 
slots went to schools previously unidentified or in the 
earliest stages of NCLB improvement. On the other 
hand, Arizona, Oregon, and South Carolina designed 
accountability systems that lowered the proportion of 
schools identified from restructuring between 2011- 
12 and 2012-13. 


While it is possible that some of these schools could 
have just barely missed their targets six consecutive 
times, in many of them, low proficiency rates in math 
and reading — either in the aggregate, or for key 
student subgroups — were a persistent problem. NCLB 
may not have adequately recognized any progress 
these schools were making, but chronically low profi- 
ciency rates should not be ignored either. Non-profi- 
cient students are likely to struggle to complete high 
school, let alone the postsecondary education and 
training that are necessary to get a well-paying job 
and support a family. 
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Table 1 
Of Title | schools in improvement under NCLB accountability, how many were 
no longer identified as priority or focus schools under waiver accountability? 


Schools from NCLB Schools Identified for 

Percentage NoLonger Improvement No Longer NCLB Improvement, 
State Identified, 2012-13 Identified, 2012-13 2011-12 
Nevada 88.00% 198 225 
Florida 81.62% 1,377 1,687 
South Carolina 79.33% 142 179 
Missouri 77TAA% 515 665 
Massachusetts 73.82% 530 718 
Oregon 73.81% 62 84 
New Jersey 73.31% 651 888 
Arizona 72.14% 233 323 
Virginia 67.16% 137 204 
Average 65.53% 
Minnesota 63.55% 190 299 
Tennessee 60.43% 142 235 
Mississippi 58.42% 59 101 
Delaware 56.76% 21 37 
Oklahoma 54.14% 85 157 
Indiana 32.20% 104 223 
Rhode Island 36.36% 12 33 
Total 4,458 6,058 


Source: State Departments of Education. 
Note: 2011-12 figures do not include non-Title | schools that were also identified for improvement under NCLB. Improvement 
is defined as School Improvement, Corrective Action, or Restructuring under NCLB, or priority and focus status under waivers. 


How to read Table 1: Oklahoma identified 157 schools for improvement in 2011-12 under its NCLB 
accountability system. 85 of them, or 54.14%, were not named priority or focus schools in 2012-13 
under its new waiver accountability system. 
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Figure 1 
Schools Identified for NCLB Improvement But Not 
Identified as Priority or Focus Schools 


: School Improvement School Improvement 
[| No Sanctions Year 1 Year 2 
Corrective Action Corrective Action Fi 
Year 1 Year 2 zi Restructuring 


Figure 1a. In 10 of 15 states, the majority of schools removed from federal interventions by 
their state's waiver had been in the earliest improvement stages under NCLB accountability. 
In eight, fewer than 10 percent of these schools had been in restructuring. 


Delaware Indiana 


Mississippi Minnesota 


2% 3% 
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Oklahoma Tennessee Virginia 
1% 4% 


Figure 1b. In five of 15 states, nearly 50 percent or more of schools removed from 
federal interventions by their state’s waiver had been in the later stages of 
improvement under NCLB accountability. 


Arizona Massachusetts Nevada 


How to interpret Figure 1: These charts show schools that had been in NCLB interventions 
in 2011-12, but were not identified as priority or focus schools in 2012-13. For instance, 
among schools in South Carolina identified for improvement under NCLB but not waivers, 
49 percent had been in restructuring, 22 percent in corrective action (both years), and 29 
percent in school improvement (both years). 
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Table 2 
Comparing the Percentage of Schools in Restructuring between Schools in 
NCLB Improvement 2011-12, Schools in Waiver Improvement 2012-13, and 
Schools No Longer Identified for Improvement 2012-13 


Of Schools In Of Schools No 
NCLB Of Priority & Focus Longer Identi- 
Improvement Schools (2012-13), fied (2012-13), 
(2011-12), Per- Percentage Percentage 
centage in Previously in Previously in 

State Restructuring Restructuring Restructuring 
South Carolina 49% 23% 49% 
Massachusetts 47% 81% 40% 
Rhode Island 45% 38% 33% 
Nevada 38% 33% 38% 
Arizona 26% 14% 26% 
Indiana 22% 12% 15% 
New Jersey 16% 39% 8% 
Missouri 16% 41% 8% 
Virginia 13% 22% Th 
Oregon 13% 3% 13% 
Minnesota 11% 15% 8% 
Tennessee Th 2% 8% 
Delaware 5% 9% 0% 
Oklahoma 3% 1% 4% 
Mississippi 3% 1% 3% 
Average 21% 22% 17% 


Source: State Departments of Education. 
Note: Improvement is defined as School Improvement, Corrective Action, or Restructuring under NCLB, or priority and focus 


status under waivers. Schools in restructuring have failed to make AYP for at least six years. 


How to read Table 2: In 2011-12, 16 percent of New Jersey's schools in improvement 
were in restructuring. Under the state's waiver in 2012-13, some of these schools become 
priority and focus schools, while others do not. Among New Jersey's priority and focus 
schools, schools from restructuring make up 39 percent of them (greater than their 16 
percent share the year before). Among the schools the state identified under NCLB but not 
waivers, schools from restructuring make up only 8 percent (less than 16 percent). In other 
words, New Jersey used its limited number of priority and focus slots to increase represen- 
tation from schools in restructuring. 


The Department made a strategic, deliberate choice 
to focus accountability efforts more narrowly on 15 
percent of Title | schools and to improve the inter- 
ventions occurring within them. While districts, and 
sometimes states, provide support to other Title | 
schools not meeting their waiver performance targets, 
it is not nearly as intensive as the activities within 
priority and focus schools. This is a trade-off with the 
relative approach, and it is worth considering what 

is happening in all of the schools previously facing 
NCLB interventions, especially those in restructuring 
and corrective action. Across 15 states, eight did not 
identify 60 percent or more of their schools in restruc- 
turing as priority or focus schools (Table 3). In three, 
Nevada, South Carolina, and Tennessee, at least three 
of every four schools previously in restructuring were 
not identified as priority or focus schools. The data tell 
a similar story for schools in corrective action, those 
schools that had missed their performance targets for 
four or five years consecutively. In eight states, more 
than half of these schools were eased from federal 
accountability efforts. And with the exception of Okla- 
homa, at least 40 percent are no longer identified as 
priority or focus schools. 


To better understand how hundreds of schools iden- 
tified by NCLB for improvement could go unidentified 
in waiver accountability systems, the specific choices 
made by the federal government and states must be 
dissected further. Chief among them is the notion 
that relative, rather than absolute, measures of school 
performance should be the deciding factor between 
school intervention and inaction. 


Right-Sizing Accountability? The 
Fifteen Percent Framework 


Given criticism that NCLB forced states to place too 
many schools in interventions and the Department's 
willingness for states to experiment with relative 
school accountability, the number of schools iden- 
tified for federal intervention in each state would be 
expected to drop between NCLB and waivers. 


In most states that is exactly what happened (Figure 
2a). The number of schools in improvement decreased 
significantly in the last year as states transitioned to 
waiver-approved accountability systems — sometimes 
to levels well below those from the earliest years of 
NCLB. Five states (Florida, Massachusetts, Nevada, 
New Jersey, and South Carolina) have identified fewer 
schools for interventions in 2012-13 than they did 

in the 2005-06 school year, reducing the scope of 
federal school improvement efforts in these states. 
While not as extreme, Arizona, Delaware, Minnesota, 
Missouri, and Virginia all designed waiver account- 
ability systems that cut the size of their federal school 
improvement efforts by at least one-third from the 
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previous year. 


But this pattern does not repeat in every state. Five 
states saw the number of schools identified for im- 
provement increase after the state received a waiver. 
Indiana, Mississippi, Oklahoma, Oregon, and Ten- 
nessee created waiver accountability systems that 
classify more schools in 2012-13 as priority or focus 
schools than were in improvement in 2011-12 under 
NCLB (Figure 2b). In fact, the number of schools facing 
interventions reached an all-time high in each state. 
Over-identification of schools for improvement was 
obviously a larger problem in some states than in 
others. This is not surprising, given that states used 
assessments, cut scores, and performance targets of 
varying rigor under NCLB. 


Waiver accountability systems are incredibly complex 
on the whole. But the complexities of states’ waivers 
rarely determine how many schools are identified 

for improvement. What mattered most were states’ 
prior choices under NCLB and the decision by the 
federal government and states to use relative rankings 
to classify priority and focus schools: the 15 percent 
framework. Many waiver accountability systems seem 
to be designed for that express purpose, beginning 
and ending with an exact number of schools that must 
be identified. For example, to classify priority schools, 
Missouri first identifies the number of schools equal 
to 5 percent of Title | schools. Then, SIG schools and 
Title I-eligible high schools with average graduation 
rates below 60 percent for three years are added 

to the category. Next, remaining Title | schools are 
ranked separately by the percentage of all students 
proficient in math and reading. Numerical ranks are 
added for each school, and the process is repeated 

to include data from the past two assessment years, 
for three years of total data. Schools are then placed 
on the priority school list in rank-order until the state 
reaches a number equal to 5 percent of Title | schools. 
The performance level of the next school on the list 
doesn't matter, even if it is indistinguishable from the 
last school included in the priority category. 


Because of the 15 percent framework, changes in the 
number of schools in improvement under states’ waiv- 
ers largely depend on what happened in that state un- 
der NCLB. NCLB’s interventions kicked in when a Title 
| school missed its targets for two consecutive years. 
Comparing the number of Title | schools in NCLB im- 
provement to the number of Title | schools statewide 
provides a better sense of the scope of each state's 
improvement system prior to waivers. Unfortunately, 
data on the number of Title | schools are not available 
for the 2011-12 or 2012-13 school years. But using 
2010-11 data as a rough estimate for the number of 
Title | schools in 2011-12, approximately 32 percent 
of Title | schools faced interventions in 2011-12 across 
the 16 states — well above the 15 percent framework 
used in states’ waivers. 
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Schools 
from Year 
Schoolsin 6NCLBIm- 
Year6NCLB provement 
Improve- No Longer 
ment, Identified, 
State 2011-12 2012-13 


5 


Nevada 


S. Carolina 88 69 
Tennessee 16 12 
Oregon 11 8 


Arizona 84 60 


Mississippi 3 2 
Mass. 341 212 
Oklahoma 5 3 


Average 


Minnesota 34 15 44.1% 
Virginia 27 10 37.0% 
New Jersey 143 51 35.7% 


R. Island 15 4 26.7% 
Delaware 2 0 0.0% 
Total 1,010 578 


Source: State Departments of Education. 


Table 3 
Of schools in restructuring or in corrective action under NCLB accountability, 
how many are no longer in improvement under waiver accountability? 


Percentage 
from Year 
6 NCLB Im- 
provement 
No Longer 
Identified, 
2012-13 


@ 
(>) 
N 


87.2% 


78.4% 


75.0% 


72.7% 


714% 


66.7% 


62.2% 


60.0% 


52.3% 


Schools in 
Year 4-5 
NCLB Im- 
provement, 
State 2011-12 
Mass. 184 
Nevada 54 
S. Carolina 42 
Missouri 200 
Arizona 74 
Oregon 12 
N. Jersey 139 
Virginia 72 
Average 
Minnesota 117 
Tennessee 36 
Mississippi 15 
R. Island 11 
Indiana 57 
Delaware 10 
Oklahoma 7 
Total 1,030 


Schools 
from Years 
4-5 NCLB 
Improve- 
ment No 
Longer 
Identified, 
2012-13 
160 
46 


31 


58 


17 


1 


688 


Percentage 
from Years 
4-5 NCLB 
Improve- 
ment No 
Longer 
Identified, 
2012-13 
87.0% 
85.2% 
73.8% 
72.0% 
67.6% 
66.7% 
64.0% 
62.5% 
57.5% 
49.6% 
47.2% 
46.7% 
45.5% 
40.4% 
40.0% 


14.3% 


Note: Improvement is defined as School Improvement, Corrective Action, or Restructuring under NCLB, or priority and focus 
status under waivers. Schools in restructuring have failed to make AYP for at least six years. Schools in corrective action have 


failed to make AYP for four or five years. 


How to read Table 3: How to read Table 3: In 2011-12, 11 Oregon schools were in year 
six of improvement. Of the 11, 8 (or 72.7 percent) were not identified as priority or focus 
schools in 2012-13. And of the 12 schools in years four or five of improvement in 2011-12, 


eight (or 66.7 percent) were not identified as priority or focus schools in 2012-13. 


But there is wide variation. Some states identified 
fewer than 15 percent of their Title | schools prior 

to waivers, while others identified far more. As Table 

4 demonstrates, the states where approximately 15 
percent or fewer Title | schools were in improvement 
under NCLB are, unsurprisingly, the same ones that in- 
creased the scope of school improvement under waiv- 
ers when they were required to intervene in 15 percent 
of schools. And those states where the proportion 

of Title | schools in improvement was well above 15 
percent are those that saw the biggest decreases, in- 
cluding Massachusetts, Nevada, New Jersey, Missouri, 
and Florida. 


The degree of change in the number of schools states 
identify for interventions before and after the tran- 
sition to waivers may show significant variance, but 
states are all moving toward the same target. As Figure 
3 makes clear, the scope of school improvement is 
noticeably converging around 15 percent of Title | 
schools. Regardless of a state’s prior or current choic- 
es in accountability design, the 15 percent target is the 
one that counts. 


But 15 percent is an arbitrary number. Perhaps 15 per- 
cent is a reasonable expectation based on the capacity 
of state education agencies, charged with leading 
school improvement efforts. Perhaps it’s too low, or 
too high, or should depend somewhat on the state's 
or school's absolute level of performance. Indeed, the 
Department of Education included two safeguards 

to try to ensure certain schools are typically priority 
or focus schools: Title |-receiving high schools with 
graduation rates below 60 percent and SIG schools.” 
Perhaps there should be additional safeguards. 


These are difficult questions, with no definitive an- 
swers. But instead of asking if states are “right-sizing” 
accountability efforts, a better question would be: 
are states identifying the right schools, regardless of 
whether they're doing more or less? Because many 
states are intervening in fewer schools, the stakes of 
identifying the right ones have gotten even higher. 
States clearly excluded many schools in restructuring 
and corrective action from their pool of priority and 
focus schools (Table 3). If they didn’t make the 15 per- 
cent cut, what kinds of schools did? 


Who are the 
Fifteen Percent? 


Within the bottom 15 percent, states could make their 
new identification processes as simple or as compli- 
cated as they liked (see Appendix A: Summary of Waiv- 
er Accountability Design Choices). They designed a 
variety of goals for schools, and sometimes — but not 
always — performance against these goals determines 
whether a school is classified as priority or focus. They 
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created elaborate indices and school grading systems, 
which are often — but not always — used to identify 
priority and focus schools. These new ranking systems 
include a variety of measures, and each state chose 
how to weight the various components. Student 
growth is typically among them, and it is used often — 
but not always — to name priority schools. The growth 
measures chosen are also of varying levels of sophisti- 
cation, reliability, and validity. And unlike NCLB, states 
occasionally — but not always — consider individual 
subgroup performance to name priority and focus 
schools. If not, they often lump these groups together 
into “super-subgroups” or pick a few subgroups on 
which to focus. 


While these choices had little impact on the number 
of schools identified, they do affect which kinds of 
schools are placed in improvement. Knapp Elementary 
was not the only school in Michigan City caught in 
limbo between NCLB and waivers. Nearby Pine 
Elementary School, a school that had received two 
straight “As” for exemplary progress, was not as for- 
tunate in the new system. Pine’s grade plummeted 

to a“D," and the school was named a focus school. 
Another Michigan City elementary school, Edgewood, 
fared even worse. After earning a string of average 
grades previously (like Knapp), Edgewood received an 
“F" and became a priority school, when it had not been 
in improvement under NCLB the year before. 


At a glance, student performance at the three schools 
does not provide many clues as to why Edgewood and 
Pine are called out for interventions under waivers but 
not NCLB, while the reverse is true for Knapp (Figure 
4). All perform below state averages. And all three had 
seen both progress and declines in student proficiency 
rates over time. 


But proficiency rates are not the only way, or perhaps 
even the best way, to judge school performance. So 
Indiana looked to other measures to determine each 
school's rating. The state added performance-based 
student subgroups and individual student growth 
measures to the mix. These changes made the dif- 
ference for Knapp, Edgewood, and Pine. Based on 
proficiency rates alone, each school earned a “D" 

and would have been a focus school. But as Figure 

5 shows, Edgewood's grade was pulled down to an 
“F" after too many of its students demonstrated low 
growth in both reading and math. Knapp’s grade was 
lifted due to particularly large numbers of students 
demonstrating high growth within a newly added su- 
per-subgroup: students in the three highest-perform- 
ing quartiles. Pine didn't receive any growth bonus or 
penalty points, so its “D" grade held. Indiana chose to 
place significant weight on student growth in its waiv- 
er, and this addition changed perceptions of quality at 
the three schools. 
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Figure 2 
Comparing the Number of Title | 
Schools in Improvement by State, 2005-2013 


Figure 2a. 11 of 16 states identify fewer schools in 2012-13 than in the previous 
year, with many states intervening in fewer schools than in 2005-06. 
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Figure 2b. Using their waivers, five of 16 states identified more schools for 
improvement in 2012-13 than under any previous year of NCLB. 
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Source: U.S. Department of Education (Title | and 2005-06 to 2010-11 school improvement data) and state Departments of 
Education (2011-12 and 2012-13 school improvement data). 

Note: Because 2011-12 and 2012-13 Title | data are not available, the percentage of Title | schools in improvement for those 
years are estimated using 2010-11 Title | data. If more recent data were available, it is likely that no state would be identifying 
fewer than 15 percent of Title | schools in 2012-13. Improvement is defined as School Improvement, Corrective Action, or 
Restructuring under NCLB, or priority and focus status under waivers. 


How to interpret Figure 3: This chart shows the percentage of Title | schools identified for improve- 
ment under NCLB and waivers from 2005-06 to 2012-13. In 2011-12, Massachusetts is one of many 
outliers. Approximately 70 percent of its Title | schools were placed in interventions based on NCLB 
accountability. But with waivers in 2012-13, there are no more outliers: all states are identifying close 
to 15 percent of Title | schools for federal intervention. 
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New growth measures made the difference in Mich- 
igan City, but every state had significant autonomy 

to design a new accountability system within the 15 
percent framework. Thus, no two waiver states are 
exactly alike. Some created systems that are more sim- 
ilar to NCLB than others. And arguably, those states 
should see less variation between the schools that are 
identified under NCLB and those that are identified by 
waivers. 


As Figure 6a shows, seven states (Arizona, Indiana, 
Mississippi, Oklahoma, Oregon, South Carolina, and 
Tennessee) used their waivers to design accountability 
systems where nearly 50 percent or more of priority 
and focus schools had not been identified in the prior 
year.’® In other words, these states seem to agree that 
NCLB accountability often missed many low-per- 
forming schools and designed their waivers to identify 
a very different group. The most common kind of 
priority and focus school in each is one that was not 
receiving any interventions the year before — schools 
like Pine and Edgewood Elementary in Michigan City. 
Further, priority and focus schools that had also been 
identified previously were often those facing the least 
serious NCLB improvement strategies. In most of these 
states, a small share of priority and focus schools had 
been in improvement for multiple years. 


On the other hand, some states used their waivers to 
redouble efforts in previously identified low-perform- 
ing schools (Figure 6b). Eight states’ new account- 
ability systems mostly identify the same schools for 
improvement. These states may have felt that NCLB 
forced them to identify too many schools, but if their 
waiver accountability systems are any indication, they 
are less convinced that NCLB identified the wrong 
ones. Virtually all of New Jersey's priority and focus 
schools had also been in need of improvement under 
NCLB, and 60 percent of them were in corrective 
action or restructuring. In Massachusetts, over eighty 
percent of priority and focus schools had been in 
restructuring, while only four percent had been in the 
first two years of school improvement. Priority and 
focus schools are distributed more evenly amongst 
NCLB’s improvement categories in other states, but 
in all of them, a majority of priority and focus schools 
faced interventions in 2011-12. Of course, every 

state also used the waiver opportunity to discontinue 
interventions in hundreds of schools, including those 
in corrective action and restructuring (Table 3). But in 
some states, newly identified schools are less fre- 
quent. In other words, misidentification of schools, like 
over-identification, may not have been as pervasive 
under NCLB as assumed. 


The shift to relative forms of school accountability 
continues to explain the school improvement picture. 
It's no surprise that the states identifying more schools 
for improvement under waivers (Figure 2b) also iden- 
tify a majority of their priority and focus schools from 


those that had not previously been in improvement 
(Figure 6a). To name 15 percent of Title | schools in 
the priority and focus categories, these states had no 
choice but to identify new schools. While each also 
removed schools, including those missing AYP for over 
five years, from interventions, these shifts are masked 
somewhat within the overall increase in their improve- 
ment activities. 


Other accountability design choices provide addition- 
al insight into which schools are included in school 
improvement efforts under waivers. Every state, like 
Indiana, that examines subgroup performance using 
the lowest-performing quartile of students as a su- 
per-subgroup is represented in Figure 6a, the states 
where the most common kind of priority and focus 
school is one that was not identified the year before. 
Under NCLB, subgroups were defined by students’ 
demographic and socio-economic characteristics, not 
their academic performance. A performance-based 
subgroup, therefore, has the potential to significantly 
alter which schools are identified. 


One reason is that there can be much less variance 
between a school’s overall performance and the per- 
formance of its lowest-performing quartile (averaged 
across all ethnic, racial, and at-risk subgroups) than 
between its overall performance and the performance 
of its lowest-performing subgroup. If a school’s only 
low-performing subgroup for accountability is special 
education students, and they represent ten percent 

of school enrollment, when added to the new su- 
per-subgroup, the performance of special education 
students would be obscured by non-special educa- 
tion students that perform better. In other words, the 
difference between the “all students” group and the 
“low-performing quartile” group is smaller than the 
difference between the “all students” group and the 
“special education students” group alone. Schools that 
would have been identified for low-performing sub- 
groups under NCLB may, therefore, be let off the hook 
under waivers. This effect does not appear to occur 
with other super-subgroups or in states that chose to 
focus on a limited number of subgroups. The decision 
to create a super-subgroup based on performance, 
rather than demographics, appears to be the distin- 
guishing factor. 


Likewise, states that heavily emphasize student growth 
over proficiency rates for identifying priority and focus 
schools might be expected to name more new schools 
for improvement under waivers, as this is a significant 
departure from how NCLB identified low-performing 
schools. Schools previously identified for low profi- 
ciency rates would be removed from improvement if 
their students demonstrate better levels of growth. 
This was the case with Knapp Elementary School, 
where growth measures increased the school's overall 
grade from a‘D' to a'C’. But across all states, the 
patterns are not as clear. Massachusetts, Minnesota, 
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Table 4 
Degree of Change in the Size of States’ School Improvement 
Activities Between NCLB and Waivers 


Percentage Percentage Number of Number of Percent 
Title | Schools Title! Schools Title | Schools Schools in Change in 
in NCLB Im- in NCLB in NCLB Waiver Im- Improvement 
provement Improvement Improvement provement Efforts, NCLB 
State 2010-11 2011-12 2011-12 2012-13 to Waivers 

Mass. 66% 71% 718 162 -77A% 
Nevada 38% 60% 225 33 -85.3% 
New Jersey 33% 60% 888 249 -72.0% 
Missouri 50% 57% 665 174 -73.8% 
Florida 39% 52% 1.687 349 “ADSM 
Minnesota 40% 35% 299 127 -57.5% 
Virginia 18% 28% 204 108 -47.0% 
Delaware 8% 22% 37 23 -37.8% 
Arizona 17% 18% 323 187 -42.1% 
S. Carolina 18% 18% 179 82 -54.2% 
Indiana 14% 16% 223 277 24.2% 
Tennessee 5% 16% 235 250 6.4% 
Oregon 11% 14% 84 94 11.9% 
R. Island 18% 14% 33 29 -12.1% 
Oklahoma 6% 13% 157 227 44.6% 
Mississippi 13% 12% 101 117 15.8% 
Average -33.5% 


Source: U.S. Department of Education (Title | and 2010-11 school improvement data) and state Departments of Education 
(2011-12 and 2012-13 school improvement data). 

Note: Italics indicate state identified less than 15 percent of its Title | schools for improvement. Red indicates negative percent 
change. 2011-12 Title | data are not available. Data on the percentage of Title | schools in improvement in 2011-12 are esti- 
mated using 2010-11 Title | data. Improvement is defined as School Improvement, Corrective Action, or Restructuring under 
NCLB, or priority and focus status under waivers. 


How to read Table 4: In 2010-11, 18% of Virginia's Title | schools were in improvement 
based on NCLB accountability. This increases to approximately 28% of Title | schools in 
2011-12 — more than the 15 percent required to be identified under waivers. Virginia's 
waiver identifies 108 priority and focus schools in 2012-13, 47 percent fewer schools in im- 
provement than the 204 identified in 2011-12. 
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Figure 4 
Comparing Student Performance at 
Three Michigan City Elementary Schools 
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Source: Indiana Department of Education 
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Figure 5 
Comparing School Report Cards for Three Michigan City Elementary Schools 


English Language Arts 


High Growth Bonus 


All Students Lowest-Performing Other Quartiles Low Growth Penalty eed pede 
Proficiency Quartile (Q1) (Q2-Q4) (All Students) : 
Penalty 
School Rate Points Rate Bonus Rate Bonus Rate Penalty 
Knapp 69.50% 1.50 32.00% 44.20% 1.00 30.40% 2.50 
Edgewood 69.10% 1.50 20.00% 30.00% 51.30% -1.00 0.50 
Pine 63.90% 1.00 22.70% 33.80% 32.20% 1.00 


High Growth Bonus 


Low Growth Penalty 


All Students Lowest-Performing Other Quartiles (All Students) — ela 
Proficiency Quartile (Q1) (Q2-Q4) Pe 
enalty 
School Rate Points Rate Bonus Rate Bonus Rate Penalty 
Knapp 74.40% 2.00 44.40% 57.00% 1.00 16.00% 3.00 
Edgewood 64.20% 1.00 20.00% 18.30% 46.30% -1.00 0.00 
Pine 67.40% 1.50 30.40% 29.40% 37.40% 1.50 


School Final 


School School Improvement Status School Final Grade Points (Ave. 
ELA & Math) 
Knapp n/a Cc 2.75 
Edgewood Priority F 0.25 
Pine Focus D 1.25 


Source: Indiana Department of Education 
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Figure 6 
Schools Identified for NCLB Improvement But Not 
Identified as Priority or Focus Schools 


: School Improvement School Improvement 
[| No Sanctions Year 1 Year 2 
Corrective Action Corrective Action Fi 
Year 1 Year 2 ier Restructuring 


Figure 6a. In seven of 15 states, nearly 50 percent or more of priority and focus schools were 
not in improvement the prior year under NCLB accountability. And in five, less than a quarter of 
priority and focus schools were previously implementing corrective actions or restructuring. 
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Figure 6b. In eight of 15 states, the majority of priority and focus schools were in improve- 
ment under NCLB the year before. In six, the majority of priority and focus schools were in 
corrective action or restructuring. And in five, the restructuring category is the most frequent 
previous distinction for priority and focus schools. 


Delaware Massachusetts Minnesota 


How to interpret Figure 6: These charts show schools that are identified as priority and 
focus schools in 2012-13, broken out by their improvement status under NCLB. For in- 
stance, among schools in Rhode Island, 24 percent of priority and focus schools were not in 
improvement the year before; 13 percent were in the first two years of school improvement, 
24 percent were in corrective action, and 39 percent were in restructuring. 
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Nevada, New Jersey, and Rhode Island all focus their 
waiver improvement efforts on schools previous- 

ly identified under NCLB despite the fact that they 
incorporated individual student growth measures 

into their accountability systems and use growth to 
designate both priority and focus schools. But Virginia 
and Missouri, which do not consider individual student 
growth at all, are also among those identifying mostly 
the same schools as NCLB. 


Identification of High Schools with Low 
Graduation Rates 


Unlike measuring subgroup performance and individ- 
ual growth, states had less discretion when it came to 
using graduation rates to identify priority and focus 
schools. Under NCLB, secondary schools were less 
likely to receive Title | funding, and were therefore, 
less likely to be labeled in need of improvement, 
compared to elementary schools, and graduation rate 
accountability was weak." If over-identification and 
misidentification of schools for improvement under 
NCLB was an issue, it is more likely to have been an 
issue for elementary schools, since fewer high schools 
or middle schools were eligible for identification in the 
first place. 


Waiver policy specifically seeks to address high school 
accountability by requiring states to use a uniform 
measure of graduation rates and allowing states to 
identify both Title-| receiving and Title l-eligible high 
schools with low graduation rates as priority or focus 
schools.”° How has this policy affected the distribution 
of schools in improvement? Even though the number 
of schools identified has fallen in most states with the 
shift to relative accountability, Table 5 demonstrates 
that within the shrinking population of identified 
schools, there is greater representation from high 
schools under waiver accountability in only half of the 
states analyzed. 


The increase in high school accountability in these 
states almost always comes at the expense of middle 
schools, rather than elementary schools. Like high 
schools, middle schools have tended to lose out to 
elementary schools for Title | funding and 


thus, mandated improvement resources under NCLB. 
According to the Department of Education, elementa- 
ry schools received 76 percent of Title | funding, com- 
pared to 14 percent for middle schools and 10 percent 
for high schools.” Massachusetts, Nevada, Oregon, 
Mississippi, Minnesota, New Jersey, and Tennessee 
may be focusing a greater share of their improvement 
efforts on high schools, but they have also scaled back 
their relative focus on middle schools. Despite the 
emphasis on high school accountability in waivers, 
changes in the distribution of schools in improvement 
are often much more dramatic for elementary and 
middle schools. 


Identification of School Improvement 
Grantees 


By definition, SIG schools would be expected to fall 

in the bottom 5 percent of all Title | schools in a state 
and be named priority schools: they have the lowest 
student proficiency, progress, and graduation rates. 
There have been three cohorts of SIG recipients 
(2010-11, 2011-12, and 2012-13), with each school 
receiving a three-year award to implement a particular 
turnaround strategy. While the Department did not 
require all SIG schools to be named priority schools, as 
some could have already improved out of the bottom 
5 percent or changed their status as a Title | school, 

it did feature SIG schools as an example of priority 
schools, and the two categories are closely aligned.” 
Thus, most SIG schools in these states would be ex- 
pected to be priority schools. 


This is the case in most states, for most SIG schools. 
As Table 6 shows, five states analyzed identify 100 
percent of SIG schools as priority or focus schools un- 
der waiver accountability, but only three (Minnesota, 
Rhode Island, and Virginia) name every SIG school as 
a priority one. While several more states exclude only 
one or two SIG schools from the priority and focus 
lists, three states failed to label half or more of them 
(Indiana, Nevada, and South Carolina.) 


It is unclear why some SIG schools were not identi- 
fied as priority or focus schools under waivers. Maybe 
states thought it was redundant to name priority 
schools that were already implementing comprehen- 
sive school reforms, like those in the SIG program. 
Perhaps these schools were no longer receiving Title 

| funding. That said, states could identify more than 
five percent of their Title | schools as priority schools, 
just as they could identify non-Title | schools. And why 
define the lowest-performing schools differently be- 
tween the SIG program and the state’s accountability 
system? How well are waivers differentiating between 
levels of school performance if SIG schools are not 
identified? States may be able to argue that these SIG 
schools were excluded from the priority and focus 
school lists for a reason, but they should have to make 
their case. 
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In sum, the consequences of states’ new waiver 
accountability system do not always follow conven- 
tional wisdom. Did NCLB force states to over-identify 
schools for improvement? Maybe, but some have used 
their waivers to identify more schools for improve- 
ment than they ever did before, because they were 
not intervening in 15 percent of schools under NCLB. 
Did NCLB make states identify the wrong schools for 
improvement and ignore some that needed support 


more urgently? Perhaps. The addition of growth 
measures or super-subgroups made a difference for 
some schools, but many states’ priority and focus 
schools are the same ones they have been working to 
improve for years. And did NCLB encourage states to 
ignore middle and high schools and focus on improv- 
ing elementary schools instead? Waiver accountability 
sometimes eased the way for a greater emphasis on 
improving high schools, but the change often oc- 
curred within a shrinking pool of identified schools, 
and many states elected to continue prioritizing ele- 
mentary schools within their waivers. 


Above all, waivers are best understood in context of 
the shift toward relative rankings for school account- 
ability. Because of the switch to a 15 percent frame- 
work within states instead of one based on an abso- 
lute standard (AYP), thousands of formerly “failing” 
schools disappeared from federal improvement in a 
single school year. These schools are not identified 

by their state's accountability systems, or if they are, 
they receive fewer specific interventions, resources, 
and supports. With growing evidence of what states’ 
waivers did—and not just a description of what they 
would do—the trade-offs between NCLB and waiver 
accountability approaches are becoming clearer. What 
remains to be seen is whether states stay the course 
through the renewal of their waivers, or if states or 
the U.S. Department of Education use these data to 
change accountability and school improvement strat- 
egies moving forward. 
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Table 5 
Changes in the Distribution of Secondary and Elementary Schools in 
Improvement from NCLB to Waiver Accountability 


Change in Distribution (Percentage Points) 


Elementary 
State High Schools Middle Schools Schools Other Schools 
Massachusetts 10.70 -4.04 -6.31 -0.36 
Nevada 8.72 -15.34 7.52 -0.90 
Oregon 8.57 -21.56 8.86 4.13 
Mississippi 449 -18.41 14.74 -0.81 
Minnesota 3.05 “0.55 0.16 “2.09 
New Jersey 2.98 =1.26 =6.07 4.35 
Tennessee 0.36 “17.99 16.02 1.61 
Indiana 0.31 2.61 -1.34 -1.58 
Florida -1.29 2.75 -0.40 “1.05 
Arizona -2.50 -1.90 5.70 “L330 
Virginia -4.07 “26.55 30.54 0.09 
Missouri -4.15 -14.40 20.17 “1.63 
South Carolina -4.31 #045 8.16 =o. L 
Oklahoma -5.42 -11.19 15.81 0.79 
Rhode Island -8.62 10.69 -2.07 0.00 
Delaware -11.40 =43.11 26.4 oe 
Average -0.16 -8.14 8.62 -0.32 


Source: State Departments of Education. 

Note: Red indicates a decrease in the distribution of schools in improvement from that level (e.g. -2.0 means high schools in 
that state represented 20% of schools in improvement under NCLB, but only 18% of priority and focus schools under waiv- 
ers). Other schools include K-8 schools, alternative schools, and other non-traditional grade configurations. Improvement is 
defined as School Improvement, Corrective Action, or Restructuring under NCLB, or priority and focus status under waivers. 


How to read Table 5: The percentage of priority and focus schools that are high schools in 
Minnesota's waiver is just over three points more than the percentage of schools in NCLB 
improvement that were high schools the year before. 
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Table 6 
The Treatment of SIG Schools under Waiver Accountability 


SIG schools % SIG 


state schools Hentifed schools thools schools school 

frcluding gccount-aspriorty/ Wantifiad identified unident- 

ability focus 

Minnesota 27 27 100.0% 100.0% 0.0% 0.0% 
R. Island 10 10 100.0% 100.0% 0.0% 0.0% 
Virginia 25 25 100.0% 100.0% 0.0% 0.0% 
N. Jersey 20 20 100.0% 95.0% 5.0% 0.0% 
Mississippi 18 18 100.0% 94.4% 5.6% 0.0% 
Florida 102 101 99.0% 98.0% 1.0% 1.0% 
Missouri 42 41 97.6% 97.6% 0.0% 2.4% 
Mass. 38 37 97.4% 94.7% 2.6% 2.6% 
Arizona 31 30 96.8% 96.8 0.0% 3.2% 
Delaware 13 11 84.6% 76.9% 7.7% 15.4% 
Oregon 19 16 84.2% 84.2% 0.0% 15.8% 
Tennessee 31 25 80.6% 71.0% 9.7% 19.4% 
Oklahoma 16 11 68.8% 68.8% 0.0% 31.3% 
Indiana 18 9 50.0% 33.3% 16.7% 50.0% 
Nevada 15 4 26.7% 26.7% 0.0% 73.3% 
S. Carolina 25 4 16.0% 16.0% 0.0% 84.0% 


Source: U.S. Department of Education (SIG data) and state Departments of Education. 
Note: Improvement is defined as School Improvement, Corrective Action, or Restructuring under NCLB, or priority and focus 
status under waivers. 


How to read Table 6: Of Florida’s 102 SIG schools, 101 of them (or 99%) are named priority 
or focus schools. More specifically, 98% of Florida's SIG schools are priority schools, 1% are 
focus schools, and 1% are neither. 
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Getting Accountability 


Tela 


Halfway through the 2013— 
14 school year, the year 
NCLB envisioned universal 
student proficiency, school 
accountability is more 
tenuous than ever. These 
systems have never been 
more complicated, and they 
have never been more dra- 
matically different from 
one state to another. And 
these differences manifest 
not only in the substance 

of states’ waiver requests, 
but also in their effects on 
schools, educators, and stu- 
dents: namely, who receives 
the attention, support, and 
resources to improve as pri- 
ority and focus schools, and 
who does not. 

And while there may be agreement that 
NCLB is broken, there is no agreement 
that waivers are working. This is in large 
part because the policy is new and so 
little is known about its implementation 


and its effect on schools and student 
learning. 


The data presented in this paper are 


some of the first to show the differences 
in school accountability between NCLB 
and waivers. Waivers allowed states to 
experiment in many ways with their 
accountability systems, but the choic- 
es states made produced few coherent 
patterns. In other words, the data raise 
more questions than they can answer. 


Still, with no reauthorization of NCLB in 
sight, waivers are here to stay, at least 
for the foreseeable future. In August 
2013 the U.S. Department of Education 
released its initial guidance for states 
seeking to renew their waivers at the 
end of the 2013-14 school year.*? The 
renewal process would be a natural 
opportunity for states and the Depart- 
ment to correct course and fix some 
elements of the waiver policy that have 
been most concerning and troublesome. 
But given updated renewal guidance 
released in November, it seems doubt- 
ful that the Department will use waiver 
renewals to drive significant changes in 
states’ plans, because not enough data 
have been gathered to determine what 
these changes should be. For instance, 
the initial renewal guidelines included 
13 guided questions, plus state-specific 
questions based on the Department's 
ongoing analysis of waiver implemen- 
tation. This process was streamlined in 
the updated guidance: states will submit 
a letter describing the highlights of their 
waivers and addressing any non- 
compliance issues identified during past 
monitoring checks. The state-specific 


42 


It's All Relative 


implementation analysis will not be a factor at this 
time. A few states will have more specific items to 
note, since they have been identified as “high-risk” 
states. But otherwise, the renewal process is limit- 
ed to concerns identified during the Department's 
already-completed Part B monitoring. 


The Department's decision to use a much more 
streamlined renewal process reflects the lack of 
information currently available about state implemen- 
tation of NCLB waivers. But the only way to determine 
if states’ waivers are improving school, educator, and 
student performance is to gather evidence, carefully 
and systematically, on what is happening and why, and 
correct course if needed. The U.S. Department of Ed- 
ucation is both responsible and ideally suited for this 
task. While it is understandable that the Department 
may need additional time and manpower to gather 
and analyze the data, it must continue these efforts 
and make the information accessible to researchers, 
analysts, journalists, policymakers, and practitioners. 
Given the critical need for better data on NCLB 
waiver implementation, the Department should 
reconsider and strengthen, where necessary, how 
it plans to provide ongoing oversight of waivers, 
including the renewal process, to ensure it is com- 
prehensive, transparent, and responsive. 


Based on this analysis of both states’ waiver requests 
and their school improvement data, there are three 
additional considerations for the Department as it 
works with states to monitor their NCLB waiver plans 
and for states as they revise and re-submit their waiv- 
er-approved accountability systems. 


1. Consequential Accountability Matters 
More than Report Card Accountability. 


Many states’ waiver requests go to inordinate lengths 
to describe their new complicated grading systems 
or performance indices. But these systems are often 
entirely removed from a school's performance goals, 
sending a signal to educators, parents, and students 
that the goals are not very realistic or important. Even 
worse, doing poorly on these report cards or ranking 
systems, or failing to meet the performance goals, can 
carry few consequences for schools. For most Title | 
schools — the 85 percent not landing on the priority 
or focus school lists like Knapp Elementary — waiver 
accountability lacks teeth. What's the point of having 
performance goals or a complicated school rating 
system if it's mostly for show? 


States should consider improving the alignment 
between their school goals, school quality mea- 
sures, and process for identifying priority and focus 
schools. The system, as a whole, should make sense 
to parents, teachers, administrators, and policymak- 
ers. And the Department should pay more attention 
to how schools are identified as priority and focus 


schools. While report card accountability is important, 
it's much more important if those report cards are used 
to identify priority and focus schools. States should jus- 
tify the measures they choose to name these schools 
and which groups of students they apply to, especially 
if the methodology is different from how all schools are 
graded or ranked for report card accountability. And 
data should be collected and published for all states to 
demonstrate how the population of schools selected 
for improvement has shifted between NCLB and NCLB 
waivers, including both the level of intervention applied 
and student achievement data, like proficiency rates 
and graduation rates. 


2. Enforce the Rules — and 
Consider Adding More 


Under waiver accountability, two kinds of schools were 
typically meant to be identified as priority and focus 
schools: SIG schools and high schools with graduation 
rates below 60 percent. Yet the data show some states 
are failing to identify many of their SIG schools, despite 
the similarity between the definitions of SIG and priority 
schools. And states are not necessarily increasing their 
relative focus on high schools within the 15 percent of 
schools they identify. Moving forward, the Department 
should ensure SIG schools are appropriately identified 
in waiver accountability systems, even in states that 
only identify their priority schools every two or three 
years. At a minimum, if a state excludes a SIG school 
from the priority and focus school lists, it should pro- 
vide justification for doing so. The Department should 
also conduct a similar check annually to monitor 
whether high schools with low graduation rates are 
consistently identified as focus and priority schools. 


The Department can make and enforce these kinds of 
rules only because there are clear definitions of Tier 

| and Tier Il SIG schools and graduation rates. The 60 
percent graduation rate threshold is meaningful only 
because all states use the same methodology and have 
comparable data. This comparability does not apply 

to measures like proficiency rates or even test scores, 
because each state sets its own definition of profi- 
ciency and uses their own test. The Department could 
hardly specify what level of performance is acceptable 
when each state has a different way of measuring it. 
This level of specificity, however, is not required when 
relative measures are used to identify priority and focus 
schools. 


But a shift to relative measures can also create perverse 
incentives for states. The 15 percent framework may 
have cut the number of schools identified for federal 
interventions in half, but it is an arbitrary cut. And arbi- 
trary cuts should be made carefully. Just look at NCLB, 
where states consistently tinkered with proficiency cut 
scores, the precision with which proficiency was mea- 
sured, and the minimum number of students required 
in a subgroup to “count.” Waivers may correct some of 
these issues, but they also create new ones. The Title | 


school at the 16th percentile could need assistance just 
as much as the school at the 15th percentile. It is worri- 
some when states’ methodology for identifying priority 
and focus schools begins with the number that must 
be identified first and foremost, rather than a consider- 
ation of what level of student achievement, or growth, 
warrants state intervention. And it’s also worrisome 
when many schools that were consistently failing to 
make AYP and implementing restructuring plans are no 
longer identified under waiver accountability systems. 


It is no secret that states have limited resources for 
school improvement, and that NCLB was becoming 

a larger strain on them each year. But in the rush to 
relieve state education agencies, did some low-per- 
forming schools fall through the cracks? The data on 
schools no longer identified, including those from 
restructuring and corrective action, show that the an- 
swer is probably yes. Are states so worried about false 
positives, identifying good schools for interventions, 
that they have increased the number of false negative 
results, failing to identify low-performing schools that 
need the help? 


NCLB waivers do not prohibit states from considering 
absolute measures of performance for accountabil- 

ity, and some chose to do so. But regardless of the 
emphasis on relative or absolute measures, as waiver 
accountability systems mature, states should examine 
which schools are being identified, and which schools 
just miss the cut. The Department should also use its 
data collection, monitoring, and research capacity to 
conduct and publish these kinds of analyses. If the 
schools that go unidentified show less improvement 
over time, states should consider increasing their 
improvement capacities and altering their meth- 
odologies for naming priority and focus schools to 
ensure certain kinds of schools are always identified. 
The good news is that as states begin to implement 
common standards and assessments, it may become 
easier for them to identify a particular level of student 
achievement—like graduation rates below 60 percent— 
that is so low as to always trigger identification as a 
priority or focus school. 


3. Think Carefully About the 15 
Percent 


Within the 15 percent framework, states changed or 
added a variety of measures to their accountability sys- 
tems in order to better identify schools. For instance, 
the Department's waiver guidance asked all states to 
use progress as one component of their new account- 
ability systems, particularly for the identification of pri- 
ority schools. But states measure progress in a variety 
of ways, some more sophisticated than others. Some 
states used individual growth measures, while others 
look at school-level progress, even comparing profi- 
ciency rates between two different student cohorts. 
The effect of adding different sorts of growth measures 
to accountability systems on the kinds of schools iden- 
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tified for improvement is unclear from this analysis, 
likely due to the lack of student achievement data. 
Further research on the issue should be conducted to 
determine whether certain kinds of growth models 
tend to better identify low-performing schools than 
others. If so, states should be encouraged to build 
their data infrastructure and discontinue the use of the 
less valid progress indicators. 


The treatment of student subgroups for accountabil- 
ity within waivers is also erratic, both between states 
and compared to NCLB. Only three of the 16 states 
analyzed here maintained subgroup accountability for 
all of the same subgroups required by NCLB to name 
focus schools. Instead, the vast majority created su- 
per-subgroups or selected a handful of subgroups for 
special attention, which likely helped some formerly 
“failing” schools escape the focus school label. This 
preliminary analysis shows that states using a low-per- 
forming super-subgroup, in particular, tend to identify 
more schools in their waivers that were not identified 
the year before, but more research is needed. Because 
these super-subgroups are new, the precise effects 

of using them to select schools for interventions are 
largely unknown. For instance, if the performance of 
a super-subgroup improves, does individual subgroup 
performance tend to improve as well? And how would 
states’ focus lists change if all of NCLB’s subgroups 
were considered? 


The Department should publish additional analyses 

to determine if states are failing to identify schools 
with significant achievement gaps or low subgroup 
achievement because they are using super-subgroups 
or selecting to examine only two or three subgroups. 
The Department should also analyze whether the kind 
of subgroup accountability used in waivers affects a 
school's progress in closing achievement gaps or im- 
proving subgroup performance. Because the guidance 
for selecting focus schools was more open-ended 
than for priority schools, this work could help tighten 
the definition of a focus school for future waivers or 
ESEA reauthorization. 


In short, if there is a link between accountability 
design choices and better outcomes for schools or 
students, those choices should be given preference 
in the waiver process. Given the near-universal crit- 
icism that NCLB failed to take into account students’ 
progress and failed to identify the right schools, states 
should be encouraged to select the best measures 
possible, including the assistance and resources 
needed to build their data and technical capacities. In 
the meantime, states should consider increasing the 
intensity and rigor of interventions in Title | schools 
that fail to meet their performance goals, but are not 
labeled as priority or focus schools. Since many of 
these schools likely have low-performing subgroups, 
this would be one way to ensure struggling schools 
are not overlooked in waiver accountability. 
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Appendix A 
Summary — Waiver Accountability Design Choices for AMOs, School 
Ratings, and Priority and Focus Schools. 


By seeking waivers from the U.S. Department of Education, states wanted accountability systems that took into 
account multiple measures of success, made more accurate distinctions between schools, and allowed for 
more nuance in identifying and improving the worst schools. States’ waivers change each of the fundamental 
components of school accountability—how they set school goals, measure school quality, and identify schools 
for interventions. Across the sixteen waiver states in this report, the incorporation of relative measures, student 
growth, and super-subgroups made for particularly dramatic changes to the way schools are identified for inter- 
ventions, with each state choosing its own process.” 


Annual Measurable Objectives 


(AMOs) 


Kind of AMOs 


Universal 


Aeizelit proficiency 


alates BEDUEE proficiency gaps by 
50% 

: Reduce proficiency gaps by 

Florida 50% 

Other: Improvement in A-F 

school grades by 2020 in- 


cluding proficiency targets 
and gap closing measures 


Indiana 


Other: Improvement on 
Composite Performance 
Index including gap closing 


Reduce proficiency gaps by 


Minnesota 50% 


Mississippi Sava proficiency gaps by 


Missouri Universal proficiency 


Other: In five years, all 
schools reach 90th per- 
centile in proficiency (as 
measured in 2010-11) 


Nevada 


Priority 
of focus 
schools 
identified 
using AMOs? 


Kind of 
school 
rating 

system 


A-F school 
grades 


A-F school 
grades 


A-F school 
grades 


index 


Composite 
index 


Composite 
index 


co | 
co | 


Composite 
index 


School Ratings 


Used to 
identify 
priority 
schools? 


Used to 
identify 
focus 
schools? 


oe 


pepe 


| ac 


Modified 


Modified 


Modified 


Student Growth 
Percentiles 


Change in Indi- 
vidual Proficien- 
cy Status 


Change in Indi- 
vidual Proficien- 
cy Status 


Student Growth 
Percentiles (bo- 
nus or penalty) 


Student Growth 
Percentiles 


Growth to tar- 
get model, indi- 
vidual growth to 
proficiency in 4 
years 


Student Growth 
Percentiles 


Super-subgroup: 


Low-performing 
quartile 


Super-subgroup: 


Low-performing 
quartile 


Super-subgroup: 


Low-performing 
quartile 


a 


subgroups 


Student Growth 
Yes Yes . 
Percentiles 


Selected sub- 
groups: low-in- 
come, ELL, & 
special educa- 
tion 


Non-priority 
high schools 
with low 
grad. rates 
auto includ- 
ed? 


Focus 


Focus Schools 


Focus schools 


schools 


identi- 


fied with 
relative 
measures? 


identified us- 
ing individual 
student growth 
measures? 


Student Growth 
Percentiles 


Change in Indi- 
vidual Proficiency 
Status 


Student Growth 
Percentiles (bo- 
nus or penalty) 


Student Growth 
Percentiles 


Growth to target 
model, individual 
growth to profi- 

ciency in 4 years 


It's All Relative 


Focus schools 
identified based on 
subgroup perfor- 
mance? 


Super-subgroup: 
Low-performing 
quartile 


Selected subgroup: 
low-income 


Super-subgroup: 
Low-performing 
quartile 


Super-subgroup: 
Low-performing 
quartile 


All subgroups 


All subgroups 


Super-subgroup: 
Low-performing 
quartile 


Selected subgroups: 
low-income, ELL, & 
special education 
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N. Jersey 


Oklahoma 


R. Island 


S. Carolina 


Tennessee 


Virginia 


Sige proficiency gaps by hisviadiex 
Other: Yearly improvement 
in AMO index, including 
proficiency rates, changes 
in proficiency, and gradu- 
ation 


A-F school 
grades 


Other: In five years, all 
schools earn ‘exceeds’ rat- 
ing, or the 90th percentile 
of performance 


Reduce proficiency gaps by Composite 
50% index 


Composite 
index 


Other: Annual 3-6 point 
increases in mean scale 
scores on assessments 


A-F school 
grades 


Other: Increase proficiency 

by 20 percent in 5 years and 
reduce proficiency gaps by 

50 percent over 8 years 


Composite 
index 


Other: Annual proficiency 
or gap closing target, and 
graduation rate target 


Priority 
of focus 
Kind of AMOs schools 
identified 
using AMOs? 


Kind of 
school 
rating 

system 


Annual Measurable Objectives 
(AMOs) 


Used to 
identify 
priority 
schools? 


Used to 
identify 
focus 
schools? 


School Ratings 


SIG 
schools 
auto- 
matically 
included? 


Some high 
schools 
with low 
grad. rates 
auto in- 
cluded? 


Student Growth 
Percentiles 
(safeguard) 


Change in Indi- 
vidual Proficien- 
cy Status 


Growth to tar- 
get model, indi- 
vidual growth to 
proficiency in 3 
years 


Student Growth 
Percentiles 


Value-Added 
Model (safe- 
guard) 


Super-subgroup: 
Low-performing 
quartile 


Super-subgroups 
minority, Select- 
ed subgroups: 
low-income, 
ELL, & special 
education 
Super-sub- 
groups: minority 
& low-income, 
ELL & special 
education 


Non-priority 
high schools 
with low 
grad. rates 
auto includ- 
ed? 


Student Growth 
Percentiles (safe- 
guard) 


Growth to target 
model, individual 
growth to profi- 

ciency in 3 years 


Student Growth 
Percentiles 


Value-Added 
Model (safeguard) 


Focus 
schools 
identi- 
fied with 
relative 
measures? 


Focus schools 
identified us- 
ing individual 
student growth 
measures? 


Focus Schools 


Selected subgroups: 
2 lowest-performing 
subgroups in school 


Selected subgroups: 
black, low-income, 
& ELL 


Selected subgroups: 
low-income, ELL, & 
special education; 
super-subgroup: 
minority 


Super-subgroups: 
minority & low-in- 
come, ELL & special 
education 


All subgroups 


Super-subgroup: 
minority 


Selected subgroups: 
black & Hispanic; 
super-subgroup: 
non-minority 
low-income, ELL, or 
special education 


Focus schools 
identified based on 
subgroup perfor- 
mance? 
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